ci: every live-gated test runs nightly (DEV-2203) - #237
Merged
Conversation
The weekly canary and monthly matrix left most live coverage cold for days: ~95 matrix combos ran twelve times a year, ~30 live specs once a week. Both now run every night, staggered — matrix at 01:00 UTC, the e2e-live canary (AI on) at 03:00 — two hours apart so they never contend for the global pool of five container slots, and deep overnight CET so real traffic is asleep. ~50 metered container sessions per night is the accepted cost; the two remaining on-demand-only tests are the token-gated share round-trip and the pkg.pr.new ref check, which need inputs a schedule cannot mint.
demtario
approved these changes
Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per Dan's review: ~130 tests skip in PR CI by design, but only 5 ran per deploy — the rest sat on a weekly canary (~30 tests) and a monthly matrix (~95 combos). Cold coverage for days at a time.
Now, every night (staggered against the 5-slot container pool)
--workers=2E2E_LIVE/E2E_BASE_URLspec: sandpack renders, all 5 Style wiring shapes, docs Vue/Angular containers + the HMR-connected oracle, share contract, row striping, preview recovery, imports, version dispatch, both AI endpointsPost-deploy
@smoke(5 tests) stays as the per-deploy heartbeat.Still on-demand only, by necessity: the authed share round-trip (needs the hand-refreshed
E2E_BROKER_TOKEN) and the pkg.pr.new container check (needs a fresh build id as a dispatch input) — a schedule can't mint either.Costs accepted with the cadence (owner decision): ~50 metered container sessions + 2 LLM calls per night, and nightly triage duty — a hosted-bundler blip now shows up as a red run the next morning instead of next Monday. Known bugs surface immediately too: expect DEV-2546 (dark striping) red in row-striping until fixed, and the matrix will answer whether DEV-2541 fixed the DEV-2547 starters.
Note
Medium Risk
No application code changes, but nightly prod runs materially increase metered container usage and LLM budget exposure, and failures will surface daily instead of weekly/monthly.
Overview
Live E2E coverage now runs every night instead of a monthly starter sweep and a weekly
e2e-livecanary.The starter compatibility matrix (
e2e-starter-matrix.yml) moves from monthly (1st at 03:00 UTC) to daily at 01:00 UTC. The live render / Sandpack canary (e2e-live.yml) moves from weekly (Monday 05:00 UTC) to daily at 03:00 UTC, still targeting prod on schedule with AI checks enabled via the existingschedulebranch of the workflow.Comments explain the two-hour stagger (01:00 vs 03:00) so both jobs do not compete for the same global five-container sandbox slots.
runner/AGENTS.md’s CI/CD table is updated to describe nightly triggers for both workflows.Reviewed by Cursor Bugbot for commit c602689. Bugbot is set up for automated code reviews on this repo. Configure here.